The energy at the Marina Bay Sands Expo and Convention Centre is always electric, but at Black Hat Asia, the air carries a specific kind of tension. When you gather thousands of the world’s most sophisticated security researchers, hackers, and practitioners under one roof, the network isn’t just infrastructure; it’s the primary target. Attendees actively probe, scan, and stress-test the infrastructure—not maliciously, but as a matter of professional instinct.
For the 2026 event, the Black Hat Network Operations Center (NOC) faced a challenge that follows directly from that reality: keeping every connection visible and every anomaly attributable, in real time, across an entire conference venue.
The Evolution of the 2026 Deployment
Building on the success of 2025, where we transitioned to Orange Pi hardware for our Enterprise Agents, 2026 marked a move toward even more distributed and resilient intelligence. This year, our focus shifted from simply knowing whether something was up or down, to understanding why, where, and—critically—what to do about it. By integrating ThousandEyes telemetry directly into the NOC’s core communication and automation workflows, we helped reduce the time from initial alert to meaningful diagnosis to seconds. This tighter loop between detection, context, and response proved critical in an environment where network conditions changed constantly, and visibility gaps could not be tolerated.
This post walks through what we built, what we observed, and how ThousandEyes helped the team navigate a week of demanding network conditions with clarity and speed.
Infrastructure
As in previous years, the Black Hat event runs in two distinct phases that drive different monitoring strategies:
-
Training days require per-room SSID monitoring. Each training room operates on its own network segment, and issues in one room must be isolated without affecting the visibility picture for adjacent rooms.
-
Conference days consolidate onto a single conference-wide SSID, shifting the focus toward aggregate performance, roaming behaviour, and upstream connectivity.
For Black Hat Asia 2026, we fielded a fleet of 25 ThousandEyes Enterprise agents distributed across the venue. ThousandEyes Enterprise Agents are software‑based sensors running on small‑form‑factor hardware that execute active probes and collect point‑in‑time network and application telemetry.
The deployment used 25 Orange Pi 5 devices, with a mix of Wi-Fi 6 (802.11ax) and Wi‑Fi 7 (802.11be) adapters, each provisioned with 256 GB of local storage—enough to support extended test cycles and buffer telemetry locally if connectivity to the platform was interrupted.
Agent Setup and Automation
Getting these agents configured, named, and connected across a venue the size of Marina Bay Sands before the first training session begins is not a trivial exercise. Our approach leaned heavily on automation.
Rather than manually configuring 25 devices during a compressed setup window, we used an automation script to push the correct Wi-Fi credentials to most of the agents remotely in a single operation, reducing a process that could have taken hours to one that took minutes and significantly reduced the risk of per-device configuration errors.
Once training days concluded, the same script was used to push the conference-wide SSID configuration, rolling the entire fleet over in a single operation.
Monitoring Coverage
Our test suite was designed to cover the full stack—from the Wi-Fi experience in each room all the way across the Internet; so that the NOC could confidently attribute any observed issue to the right layer without guessing.
The goal wasn’t to control the network; it was to understand how the network was behaving in real time. This distinction is powerful.
Monitoring and Visibility During the Event
Throughout the event, the NOC relied on a combination of active testing, passive monitoring, and platform‑level insights to maintain visibility across the venue and its upstream dependencies.
-
Active Network and Application Tests: The following active tests continuously generated real-time telemetry from each ThousandEyes Enterprise agent:
-
Internal throughput tests: A 50 MB download test from each agent, used as a per-room signal for Wi-Fi and SSID health.
-
HTTP and DNS availability tests: Probes targeting Cisco Umbrella (the DNS security service used to filter and monitor web traffic at the event) testing both the internal DNS resolver and external Internet reachability.
-
Bidirectional Network tests to public cloud providers: Network tests between venue agents and AWS, Azure, and Google Cloud, reflecting the cloud platform performance. These are most heavily used by training labs and conference demos.
-
-
Passive Routing Visibility
-
In addition to active testing, we monitored changes in how Internet traffic was routed to and from the venue, watching for upstream path shifts that could impact performance or availability before issues became visible to users.
-
-
Regional and Ecosystem Context
-
Finally, Internet Insights provided macro‑level visibility into network and application disruptions affecting key SaaS providers, ISPs, and DNS services across the region. While not a test itself, this view added critical external context that the venue’s own measurements could not capture.
-
Incidents and Observations
A Quiet Registration Desk That Wasn't
In the early hours of the second conference day—well before staff arrived on site—ThousandEyes had already flagged a problem at the registration zone. When we reviewed the overnight telemetry that morning, the cause had already been characterized. On the surface, everything looked fine.
But ThousandEyes was already telling a different story:
While the wired sensor showed no issues, the wireless agent stationed near the main registration zone flagged elevated network latency and intermittent high HTTP response times against the badge application endpoint. The anomaly was subtle, but the telemetry was clear: something was degrading the wireless path.
On-site inspection revealed that the dual antennas of the wireless sensor had shifted from their intended vertical orientation and were lying horizontally, resting directly against the aluminium structural extrusion of the registration desk framing. This unintended contact with a dense, conductive metallic surface had resulted in antenna detuning: the metallic surface disrupted the antenna's ability to radiate signal cleanly, degrading the connection quality between the sensor and the access point. Weaker signal means more retransmissions, and retransmissions are exactly what elevated latency and HTTP response time variability look like in the telemetry.
The antennas were repositioned within minutes. The wireless telemetry returned to baseline almost immediately, with HTTP response times and jitter realigning with the wired agent's readings.
The Common Sync: When Three Clouds Blinked at Once
Unrelated to the registration desk issue, and on a different day of the event, the monitoring picked up something that took a moment to make sense of—a synchronized packet loss event hitting AWS, Azure, and GCP's Singapore regions simultaneously. The precision of the timing—down to the same two-minute window—provided a textbook example of how multi-cloud observability can rapidly isolate a fault domain to shared local congestion rather than any individual cloud provider.
What the Timing Revealed: Temporal Correlation
The most telling signal wasn't the magnitude of the loss—it was synchronicity. All three ThousandEyes data show the red loss spike originating in the same 01:48 GMT+1 bucket, with matching ramp-up and decay curves—meaning the loss didn't just start simultaneously; it built and cleared at the same rate across all three providers, a pattern that would be statistically implausible if the causes were independent.
Figure 6: AWS Asia test showing synchronized packet loss.
Apr 24 01:48-01:50 GMT+1: note the spike starting in the same window as Azure and GCP.
Figure 7: Azure Asia test showing the same loss window.
Apr 24 01:48-01:50 GMT+1: the spike rises and clears in step with AWS and GCP.
Figure 8: GCP Asia test confirming shared packet loss timing.
Apr 24 01:48-01:50 GMT+1: the matching spike and decay curve confirms synchronized impact.
Think of it this way: if the problem were inside AWS, only AWS would show it. The fact that all three failed together means that the cause of the failure must be something they all share: the path from the venue to the Internet, not inside any provider's network. That immediately ruled out cloud-specific causes and pointed directly at the venue's own network or its first-hop connection to the Internet.
The congestion cleared on its own before it was ever formally escalated. Measured in impact, it was a minor blip—a handful of minutes, single-digit aggregate loss; no tickets were raised. Measured in insight, it was anything but minor. This is the quiet power of multi-cloud observability: it turns three separate mysteries into one clear answer, often before anyone needs to ask the question.
The Slack Bot - ThousandEyes MCP Integration
One of the more notable operational changes at Black Hat Asia 2026 was how the NOC team queried ThousandEyes data during live incidents. Previously, the flow was passive: ThousandEyes pushed the alerts and engineers had to context-switch to the UI to investigate. This year, we tried closing that gap by testing a Slack Bot backed by the ThousandEyes MCP (Model Context Protocol) server, powered by a locally hosted LLM model.
Why Local?
The ThousandEyes MCP server processes detailed operational telemetry: agent configurations, path visualizations, BGP paths, SSID details, and live network data that collectively provide a comprehensive view of the venue's infrastructure. Running the model locally contained all inference within the NOC's infrastructure boundary—network topology, test configurations, and telemetry data never left the controlled perimeter.
From Dashboards to Conversational Observability
While this Slack Bot integration was tested as a controlled capability rather than a scaled production workflow, it showed an important direction for AI-assisted network observability. By combining ThousandEyes telemetry, MCP-based tool access, and an approved LLM runtime, engineers could query operational data in natural language from the same chat tool where incident coordination was already happening.
Instead of switching between team messaging apps, dashboards, test views, path visualizations, and API outputs, the NOC could ask questions such as “which tests are failing?”, “which agents are affected?”, or “summarize the latest packet loss across cloud providers” and receive a contextual response directly in the conversation. The LLM becomes an interface layer that translates intent into the right observability query, retrieves the relevant telemetry, and summarizes the evidence for human review.
The broader implication is clear: as LLM tools mature and integrate safely with observability platforms, network operations can move toward a future where insight is available where teams already collaborate. Dashboards and detailed views remain essential, but the first layer of diagnosis can happen inside a chat workflow, reducing context switching and helping teams move from alert to understanding much faster.
About Black Hat
Black Hat is the cybersecurity industry’s most established and in-depth security event series. Founded in 1997, these annual, multi-day events provide attendees with the latest in cybersecurity research, development, and trends. Driven by the needs of the community, Black Hat events showcase content directly from the community through briefing presentations, training courses, summits, and more. Black Hat is an event series where all career levels and academic disciplines convene to collaborate and network, with events held in the United States, Canada, Europe, the Middle East and Africa, and Asia. For more information, please visit BlackHat.com.
Looking Ahead
As Black Hat Asia 2026 wraps up, the lesson is clear: in an era of hyper-connectivity, visibility is essential to stay ahead of the chaos. Whether it’s managing automated bot integrations or troubleshooting high-density wireless congestion, having a clear view of critical network paths help ensure the NOC stays in control.
Cisco and the Cisco logo are trademarks or registered trademarks of Cisco and/or its affiliates in the U.S. and other countries. To view a list of Cisco trademarks, go to this URL: www.cisco.com/go/trademarks. Third-party trademarks mentioned are the property of their respective owners. The use of the word partner does not imply a partnership relationship between Cisco and any other company.
